随机梯度下降（SGD）是现代机器学习（ML）系统的基石。尽管具有其计算效率，但SGD仍需要随机数据访问，这些数据访问在依赖块可调地理的二级存储的系统中实现效率低下，例如HDD和SSD，例如TensorFlow/Pytorch和DB ML系统，而不是大文件。为了解决这种阻抗不匹配，已经提出了各种数据改组策略，以平衡SGD的收敛速率（有利于随机性）及其I/O性能（有利于顺序访问）。在本文中，我们首先对现有数据改组策略进行系统的实证研究，该研究表明，所有现有策略都有改进的空间 - 它们都在I/O性能或融合率方面受苦。考虑到这一点，我们提出了一种简单但新颖的分层数据改组策略Corgipile。与现有的策略相比，Corgipile避免了完整的数据洗牌，同时保持SGD的可比收敛速度，就好像执行了完整的混音一样。我们对Corgipile的融合行为提供了非平凡的理论分析。我们通过在新的CorgipileDataSet API中设计新的平行/分布式洗牌操作员来进一步将Corgipile整合到Pytorch中。我们还通过介绍具有优化的三个新的物理运营商，将Corgipile集成到PostgreSQL中。我们的实验结果表明，Corgipile可以与全面的SGD达到可比的收敛速率，以实现深度学习和广义线性模型。对于ImageNet数据集的深度学习模型，Corgipile比带有完整数据洗牌的Pytorch快1.5倍。对于具有线性模型的INDB ML，在HDD和SSD上，Corgipile的Corgipile比两个最先进的IN-DB ML系统（Apache Madlib和Bismarck）快1.6 x-12.8倍。

沟通压缩是现代分布式学习系统的至关重要技术，可以减轻其在较慢的网络上的交流瓶颈。尽管最近对数据并行式训练的梯度压缩进行了深入的研究，但压缩了通过管道并行性训练的模型的激活仍然是一个空旷的问题。在本文中，我们提出了AC-SGD，这是一种新型的激活压缩算法，用于在慢速网络上进行通信有效的管道并行性训练。 AC-SGD与以前的激活压缩方面的努力不同，而不是直接压缩激活值，而是压缩激活的变化。这使我们能够首次向我们的知识表明，仍然可以实现$ o（1/\ sqrt {t}）$收敛速率，即激活压缩的非convex目标，而无需对梯度做出假设无偏见对于具有非线性激活功能的深度学习模型不符合。然后，我们证明AC-SGD可以有效地优化和实施，而无需额外的端到端运行时开销。我们将AC-SGD评估为微调语言具有高达15亿个参数的模型，将激活压缩至2-4位。AC-SGD在较慢的网络中可提供高达4.3倍的端到端速度，而无需牺牲模型质量。此外，我们还表明，AC-SGD可以与最先进的梯度压缩算法结合使用，以启用“端到端通信压缩：机器之间的所有通信，包括模型梯度，远期激活和后退梯度压缩为较低的精度。这提供了高达4.9倍的端到端加速，而无需牺牲模型质量。

训练基金会模型（例如GPT-3和Palm）可能非常昂贵，通常涉及数以万计的GPU连续运行数月。这些模型通常经过专门的群集培训，这些群集具有快速，均匀的互连，并使用精心设计的软件系统来支持数据并行性和模型/管道并行性。这样的专用集群可能是昂贵且难以获得的。我们可以相反，可以利用更大量的分散，异质和较低的互连计算？先前的工作研究了可以纯粹以数据并行方式训练的相对较小模型的异质，分散的设置重点。模型平行基础模型培训（例如威震天）的最先进的方案仅考虑均匀的数据中心设置。在本文中，我们介绍了第一个研究大型基础模型的研究，该模型在异质网络上的去中心化制度中进行了模型并行性。我们的主要技术贡献是一种调度算法，该算法将不同的计算“任务”在培训基础模型中分配给通过缓慢的异质网络连接的一组分散的GPU设备。我们提供了正式的成本模型，并进一步提出了一种有效的进化算法，以找到最佳分配策略。我们进行了广泛的实验，这些实验代表了使用现实世界网络测量模拟的地理分布设备进行学习的不同方案。在最极端的情况下，在跨越3大洲的8个不同的城市中，我们的方法比以前的最新培训系统（Megatron）快4.8倍。

基于深度学习的模型占主导地位的生产推荐系统的当前景观。此外，近年来目睹了模型规模的指数增长 - 从谷歌的2016年模型，最新的Facebook的型号有10亿个参数，具有12万亿参数。型号容量的每次跳跃都有显着的质量增强，这使我们相信100万亿参数的时代即将来临。然而，即使在工业规模数据中心内，这些模型的培训也在挑战。这种困难是从训练计算的惊人的异质性继承 - 模型的嵌入层可以包括总模型尺寸的99.99％，这是极其内存密集的;虽然其余的神经网络越来越多地计算密集型。为支持培训此类巨大模式，迫切需要有效的分布式培训系统。在本文中，我们通过仔细共同设计优化算法和分布式系统架构来解决这一挑战。具体而言，为了确保培训效率和训练精度，我们设计一种新型混合训练算法，其中嵌入层和密集的神经网络由不同的同步机制处理;然后，我们构建一个名为Persia的系统（短暂的并行推荐培训系统，其中包含混合加速），以支持这种混合培训算法。理论上的示范和实证研究均达到100万亿参数，以证明了波斯的系统设计和实施。我们将Pensia公开使用（在https://github.com/persiamml/persia），以便任何人都能够以100万亿参数的规模轻松培训推荐模型。

近年来，目睹了分布式数据并行培训的越来越多的系统列表。现有系统很大程度上适合两个范例，即参数服务器和MPI样式的集体操作。在算法方面，研究人员提出了广泛的技术，以通过系统弛豫降低通信：量化，分散和通信延迟。然而，大多数情况下，如果不是全部，现有系统仅依赖于标准的同步和异步随机梯度（SG）的优化，因此不能利用机器学习社区最近发展的所有可能的优化。鉴于该系统和理论的当前景观之间的新出现差距，我们构建了一个MPI式通信库，提供了一种基元的集合，这既灵活又模块化，以支持分布式的最先进的系统松弛技术训练。 BAGUA提供了这种设计，拥有巨大的实现和扩展各种最先进的分布式学习算法的能力。在具有多达16台机器（128个GPU）的生产群集中，BAGUA可以在端到端培训时间内优于Pytorch-DDP，Horovod和ByTeps，在各种任务范围内的重大边缘（最多2次）。此外，我们进行严格的权衡探索，表明不同的算法和系统放松在不同的网络条件下实现了最佳性能。

Graph Neural Networks (GNNs) have shown satisfying performance on various graph learning tasks. To achieve better fitting capability, most GNNs are with a large number of parameters, which makes these GNNs computationally expensive. Therefore, it is difficult to deploy them onto edge devices with scarce computational resources, e.g., mobile phones and wearable smart devices. Knowledge Distillation (KD) is a common solution to compress GNNs, where a light-weighted model (i.e., the student model) is encouraged to mimic the behavior of a computationally expensive GNN (i.e., the teacher GNN model). Nevertheless, most existing GNN-based KD methods lack fairness consideration. As a consequence, the student model usually inherits and even exaggerates the bias from the teacher GNN. To handle such a problem, we take initial steps towards fair knowledge distillation for GNNs. Specifically, we first formulate a novel problem of fair knowledge distillation for GNN-based teacher-student frameworks. Then we propose a principled framework named RELIANT to mitigate the bias exhibited by the student model. Notably, the design of RELIANT is decoupled from any specific teacher and student model structures, and thus can be easily adapted to various GNN-based KD frameworks. We perform extensive experiments on multiple real-world datasets, which corroborates that RELIANT achieves less biased GNN knowledge distillation while maintaining high prediction utility.

To generate high quality rendering images for real time applications, it is often to trace only a few samples-per-pixel (spp) at a lower resolution and then supersample to the high resolution. Based on the observation that the rendered pixels at a low resolution are typically highly aliased, we present a novel method for neural supersampling based on ray tracing 1/4-spp samples at the high resolution. Our key insight is that the ray-traced samples at the target resolution are accurate and reliable, which makes the supersampling an interpolation problem. We present a mask-reinforced neural network to reconstruct and interpolate high-quality image sequences. First, a novel temporal accumulation network is introduced to compute the correlation between current and previous features to significantly improve their temporal stability. Then a reconstruct network based on a multi-scale U-Net with skip connections is adopted for reconstruction and generation of the desired high-resolution image. Experimental results and comparisons have shown that our proposed method can generate higher quality results of supersampling, without increasing the total number of ray-tracing samples, over current state-of-the-art methods.

Panoptic Part Segmentation (PPS) unifies panoptic segmentation and part segmentation into one task. Previous works utilize separated approaches to handle thing, stuff, and part predictions without shared computation and task association. We aim to unify these tasks at the architectural level, designing the first end-to-end unified framework named Panoptic-PartFormer. Moreover, we find the previous metric PartPQ biases to PQ. To handle both issues, we make the following contributions: Firstly, we design a meta-architecture that decouples part feature and things/stuff feature, respectively. We model things, stuff, and parts as object queries and directly learn to optimize all three forms of prediction as a unified mask prediction and classification problem. We term our model as Panoptic-PartFormer. Secondly, we propose a new metric Part-Whole Quality (PWQ) to better measure such task from both pixel-region and part-whole perspectives. It can also decouple the error for part segmentation and panoptic segmentation. Thirdly, inspired by Mask2Former, based on our meta-architecture, we propose Panoptic-PartFormer++ and design a new part-whole cross attention scheme to further boost part segmentation qualities. We design a new part-whole interaction method using masked cross attention. Finally, the extensive ablation studies and analysis demonstrate the effectiveness of both Panoptic-PartFormer and Panoptic-PartFormer++. Compared with previous Panoptic-PartFormer, our Panoptic-PartFormer++ achieves 2% PartPQ and 3% PWQ improvements on the Cityscapes PPS dataset and 5% PartPQ on the Pascal Context PPS dataset. On both datasets, Panoptic-PartFormer++ achieves new state-of-the-art results with a significant cost drop of 70% on GFlops and 50% on parameters. Our models can serve as a strong baseline and aid future research in PPS. Code will be available.

An increasing number of public datasets have shown a marked clinical impact on assessing anatomical structures. However, each of the datasets is small, partially labeled, and rarely investigates severe tumor subjects. Moreover, current models are limited to segmenting specific organs/tumors, which can not be extended to novel domains and classes. To tackle these limitations, we introduce embedding learned from Contrastive Language-Image Pre-training (CLIP) to segmentation models, dubbed the CLIP-Driven Universal Model. The Universal Model can better segment 25 organs and 6 types of tumors by exploiting the semantic relationship between abdominal structures. The model is developed from an assembly of 14 datasets with 3,410 CT scans and evaluated on 6,162 external CT scans from 3 datasets. We rank first on the public leaderboard of the Medical Segmentation Decathlon (MSD) and achieve the state-of-the-art results on Beyond The Cranial Vault (BTCV). Compared with dataset-specific models, the Universal Model is computationally more efficient (6x faster), generalizes better to CT scans from varying sites, and shows stronger transfer learning performance on novel tasks. The design of CLIP embedding enables the Universal Model to be easily extended to new classes without catastrophically forgetting the previously learned classes.

This paper illustrates the technologies of user next intent prediction with a concept knowledge graph. The system has been deployed on the Web at Alipay, serving more than 100 million daily active users. Specifically, we propose AlipayKG to explicitly characterize user intent, which is an offline concept knowledge graph in the Life-Service domain modeling the historical behaviors of users, the rich content interacted by users and the relations between them. We further introduce a Transformer-based model which integrates expert rules from the knowledge graph to infer the online user's next intent. Experimental results demonstrate that the proposed system can effectively enhance the performance of the downstream tasks while retaining explainability.